Papers with attack success rate
Agent vs. Agent: Automated Data Generation and Red-Teaming for Custom Agentic Workflows (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing red-teaming frameworks like AgentHarm use static prompts and hardcoded toolsets . |
| Approach: | They propose a red-teaming framework that generates adversarial tasks and evaluation functions tailored to arbitrary toolsets and uses iterative prompt refinement with self-reflection to develop more effective attacks. |
| Outcome: | The proposed approach achieves 162% increase in attack success rate on o4-mini and 86% success on gemini 2.5 Pro. |
Dagger Behind Smile: Fool LLMs with a Happy Ending Story (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have attracted significant attention from jailbreak attacks . existing manual designs are either easily detectable or require intricate interactions with LLMs. |
| Approach: | They propose a happy ending attack that wraps up a malicious request in a scenario template . |
| Outcome: | The proposed attack wraps up a malicious request in a scenario template involving a positive prompt formed mainly via a happy ending, fooling LLMs into jailbreaking either immediately or at a follow-up malicious request. |
TrojFSP: Trojan Insertion in Few-shot Prompt Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Prompt tuning on a few data samples presents security issues, e.g., Trojan attacks. |
| Approach: | They propose a method to transfer established data poisoning attacks directly to few-shot prompt tuning, a technique to address the poisoned imbalance issue. |
| Outcome: | The proposed method achieves an ASR of over 99% while maintaining negligible decreases in CDA. |
CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing (2026.acl-long)
Copied to clipboard
Rui Wang, Junda Wu, Yu Xia, Tong Yu, Ruiyi Zhang, Ryan A. Rossi, Subrata Mitra, Lina Yao, Julian McAuley
| Challenge: | Existing Large Language Models exhibit critical vulnerability to indirect prompt injection attacks, where instructions injected within in the prompt context can override the user's intent. |
| Approach: | They propose a neural pruning algorithm that prunes neurons associated with instruction-following during KV cache encoding of the prompt context. |
| Outcome: | The proposed approach significantly reduces the attack success rate while preserving the model's ability to follow user instructions. |
Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm (2025.findings-naacl)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning (PEFT) is a key training strategy for large language models, but its reliance on fewer trainable parameters poses security risks, such as task-agnostic backdoors. |
| Approach: | They propose a method that amplifies benign neurons within PEFT layers and penalizes the influence of trigger tokens. |
| Outcome: | The proposed method reduces the success rate of state-of-the-art task-agnostic backdoors by 83.6%. |
Intrinsic Model Weaknesses: How Priming Attacks Unveil Vulnerabilities in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significant impact on various industries and societal functions due to advanced instruction-following capabilities. |
| Approach: | They developed and tested novel attack strategies on popular LLMs to expose their vulnerabilities in generating harmful content. |
| Outcome: | The proposed attacks achieved an ASR of 100% on open-source models, including Meta’s Llama-3.2, Google’s Gemma-2, Mistral’s Mistral-NeMo, Falcon’s Falcon-mamba, Apple’s DCLM, Microsoft’s Phi3, and Qwen’s Qwend2.5, among others. |
Where to Attack: A Dynamic Locator Model for Backdoor Attack in Text Classifications (2022.coling-1)
Copied to clipboard
| Challenge: | BackDoor Attack (BDA) study aims to train a poisoned model with clean data and some trigger-embedded instances to perform normally on normal inputs. |
| Approach: | They propose to train a poisoned model with clean and poisonest inputs . they propose to use triggers to predict those poisonets as target labels . |
| Outcome: | The proposed model can predict P2P dynamically without human intervention. |
SafetyQuizzer: Timely and Dynamic Evaluation on the Safety of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used to evaluate the safety of their users . however, evaluation questions in current benchmarks are too straightforward and difficult to update with practical relevance due to their lack of correlation with real-world events. |
| Approach: | They propose a question-generation framework to evaluate the safety of LLMs in the Chinese context. |
| Outcome: | The proposed framework reduces decline rate while maintaining similar attack success rate. |
SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to design sophisticated instructions for the LLM to follow, or rely on multiple iterations, could hinder the performance and efficiency of jailbreaks. |
| Approach: | They propose a simple assistive task linkage paradigm which masks harmful keywords within malicious queries and uses a masked language model task to encode the semantics of the mangled keywords. |
| Outcome: | The proposed paradigm can effectively circumvent LLM safeguards and elicit harmful responses. |
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show superior cross-model transferability . Prompt injection attacks are among the most critical threats . |
| Approach: | They propose an activations-guided prompt injection attack framework to address the impracticality of existing white-box/gray-box methods and the poor transferability of black-box approaches. |
| Outcome: | The proposed framework achieves 49.6% success rate and 34.6% improvement over human-crafted prompts on five mainstream LLMs. |
DA3: A Distribution-Aware Adversarial Attack against Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent attacks have shown that adversarial examples have a different data distribution than the original examples, reducing their effectiveness under detection methods. |
| Approach: | They propose a distribution-aware adversarial attack method that considers the distribution shifts of adversarials to improve attacks’ effectiveness under detection methods. |
| Outcome: | The proposed method improves the effectiveness of adversarial examples under detection methods and integrates both ASR and detectability. |
ValCAT: Variable-Length Contextualized Adversarial Transformations Using Encoder-Decoder Language Model (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing word-level approaches to attack text are limited to a single word . existing methods ignore interactions between consecutive words, resulting in one-to-one attacks . |
| Approach: | They propose a black-box attack framework that misleads the language model by applying variable-length contextualized transformations to the original text. |
| Outcome: | The proposed framework outperforms existing methods on classification and inference tasks. |
ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing red-teaming approaches prioritize high attack success rate, resulting in high-perplexity prompts. |
| Approach: | a new method uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate. |
| Outcome: | ASTPrompter achieves 5.1 times higher attack success rate on Llama-8.1B . low-perplexity attacks are more difficult to filter and more likely to arise during benign usage . |
Attack Named Entity Recognition by Entity Boundary Interference (2024.lrec-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a cornerstone natural language processing task . despite its robustness, studies on its robustity are lacking. |
| Approach: | They propose a one-word modification NER attack that strategically inserts a new boundary into the sentence and triggers the model to make a wrong recognition. |
| Outcome: | The proposed method is effective on English and Chinese models with 70%-90% success rate. |
DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agent (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for detecting advanced backdoors is proposed to bypass safety audits. |
| Approach: | They propose a backdoor implantation strategy that introduces dynamic encryption to bypass safety audits. |
| Outcome: | The proposed method achieves an attack success rate approaching 100% while maintaining a detection rate of 0%. |
ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Attaching suffixes to harmful instructions can hack the defense of Large language models (LLMs) However, due to the unreadable of adversarial suffix, it can be relatively easily penetrated by common defense methods such as perplexity filters. |
| Approach: | They propose an algorithm to embed adversarial suffixes into coherent and understandable text to attack Large language models (LLMs) using a Advbench dataset. |
| Outcome: | The proposed approach reduces the computation time of adversarial suffixes and achieves a much better attack success rate than existing techniques. |
ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger (2024.naacl-long)
Copied to clipboard
| Challenge: | Textual backdoor attacks are increasingly challenging to detect due to the use of advanced generative models such as GPT-4. |
| Approach: | They propose a framework that harnesses advanced generative models to execute stealthier backdoor attacks on text classifiers. |
| Outcome: | The proposed framework achieves state-of-the-art attack success rate of 97.35% over four sentiment classification tasks and four human cognition stealthiness tests. |
Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | et al., 2021) show that instruction models can be trained on crowdsourced datasets with task instructions to achieve superior performance. |
| Approach: | They examine security concerns of emergent instruction tuning paradigm that models are trained on crowdsourced datasets with task instructions to achieve superior performance. |
| Outcome: | The proposed model can achieve 90% success rate across four commonly used datasets. |
SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing red-teaming frameworks do not cover all the risks associated with arbitrary black-box LLMs. |
| Approach: | They propose a generic red-teaming framework for arbitrary black-box LLM agents that iteratively constructs and refines model-based adversarial attacks based on the execution trajectories of former attempts. |
| Outcome: | The proposed model improves attack success rate by 100%, surpassing the 671B Deepseek-R1 model. |
Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have gained increasing attention for their capacity to generate harmful content. |
| Approach: | They propose a scalable evolution framework to evolve red teaming prompts across breadth and depth dimensions, facilitating automatic generation of numerous high-quality and diverse red team prompts. |
| Outcome: | The proposed framework surpasses existing red teaming methods on attack success rate and diversity. |
Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience (2025.emnlp-main)
Copied to clipboard
Xi Wang, Songlei Jian, Shasha Li, Xiaopeng Li, Bin Ji, Ma Jun, Xiaodong Liu, Jing Wang, Jianfeng Zhang, Jie Yu, Feilong Bao, null Wangbaosheng
| Challenge: | Existing methods to generate human-aligned content with a “jailbreak prompt” are inefficient and repetitive, causing inefficiency and a lack of experience. |
| Approach: | They propose a framework that integrates past attack experiences to aid current jailbreak attempts. |
| Outcome: | The proposed framework improves both attack effectiveness and efficiency compared to the current black-box jailbreak method. |
Intention Analysis Makes LLMs A Good Jailbreak Defender (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to align large language models with human values overlook the intrinsic nature of jailbreaks, which limits their effectiveness in complex scenarios. |
| Approach: | They propose a simple yet highly effective defense strategy, i.e., Intention Analysis (IA). They show that IA suppresses LLM’s tendency to follow jailbreak prompts, thereby enhancing safety. |
| Outcome: | The proposed strategy reduces harmfulness of LLMs and outperforms GPT-3.5 in attack success rate. |
Corpus Poisoning via Approximate Greedy Gradient Descent (2025.findings-acl)
Copied to clipboard
| Challenge: | Dense retrievers have been shown to be vulnerable to corpus poisoning attacks . a malicious user injects a small fraction of adversarial passages into the retrieval corpus to trick the system into returning these passages among the top-ranked results for a broad set of user queries. |
| Approach: | They propose an attack on dense retrieval systems based on the widely used HotFlip method for efficiently generating adversarial passages. |
| Outcome: | The proposed method achieves a high success rate on several datasets and using several retrievers, and can generalize to unseen queries and new domains. |
System Prompt Hijacking via Permutation Triggers in LLM Supply Chains (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are rapidly transforming the landscape of artificial intelligence due to the substantial resources required for training. |
| Approach: | They propose a post-deployment attack that bypasses system prompts to compromise models . they introduce Precise Activation Guarding and Unit Deviation Sampling to protect against attack . |
| Outcome: | The proposed attack bypasses system prompts, enabling unrestricted model outputs and safety violations. |
Maximum Entropy Loss, the Silver Bullet Targeting Backdoor Attacks in Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing backdoor defense paradigms focus on detecting and removing poisoned samples at pre-training or inference time. |
| Approach: | They propose a new approach where the backdoor attack is directly reversed by incorporating maximum entropy loss into training to neutralize the minimal cross-entropiness loss fine-tuning on poisoned data. |
| Outcome: | The proposed model significantly lowers the attack success rate on classification tasks and reduces the risk of backdoor attacks on clean data. |
Backdoor Attacks on Multilingual Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that multilingual machine translation systems are vulnerable to backdoor attacks through data poisoning. |
| Approach: | They propose to investigate the security of multilingual machine translation systems by exposing poisoned data into low-resource languages to cause malicious translations. |
| Outcome: | The proposed method achieves an average of 20% success rate in attacking high-resource languages. |
On the Vulnerability of Text Sanitization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing reconstruction attacks on text sanitization are not able to accurately assess their effectiveness. |
| Approach: | They propose to use ASR to measure the effectiveness of reconstruction attacks to evaluate sanitization performance. |
| Outcome: | The proposed reconstruction attacks achieve a 46.4% improvement in ASR over the state-of-the-art baseline with a privacy budget of =4.0 on the SST-2 dataset. |
Evaluating the Validity of Word-level Adversarial Attacks with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing adversarial examples can generate invalid adversarials due to significant changes in semantic meanings compared to their originals. |
| Approach: | They propose to use a large language model to evaluate adversarial examples by semantic constraints. |
| Outcome: | The proposed method can generate valid adversarial examples even when they are not equipped with semantic constraints. |
SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Despite advances in large language models, they face substantial challenges in terms of safety. |
| Approach: | They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks. |
| Outcome: | The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs. |
Reevaluating Adversarial Examples in Natural Language (2020.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art adversarial examples lack a common definition of what constitutes success . human surveys show that to preserve semantics, we need to increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences. |
| Approach: | They propose a unified definition of what constitutes a successful adversarial example . they propose four categories of constraints that are used to define adversarials . |
| Outcome: | The proposed framework is based on the outputs of two state-of-the-art synonym substitution attacks. |
Jailbreak Attack Initializations as Extractors of Compliance Directions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Safety-aligned LLMs respond to prompts with compliance or refusal, each corresponding to distinct directions in the model’s activation space. |
| Approach: | They propose an initialization framework that aims to project unseen prompts further along compliance directions. |
| Outcome: | The proposed initialization framework achieves an increased attack success rate and reduced computational overhead, highlighting the fragility of safety-aligned LLMs. |
Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Ciphers (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing jailbreak methods struggle to balance effectiveness with robustness against adaptive safety mechanisms. |
| Approach: | They propose a novel approach that targets Large Reasoning Models through an adaptive encryption pipeline designed to overwhelm their reasoning capabilities. |
| Outcome: | The proposed approach achieves an attack success rate of 85.6% on OpenAI GPT-o4-mini, outperforming state-of-the-art baselines by a significant margin of 17.2%. |
TABS: Efficient Textual Adversarial Attack for Pre-trained NL Code Model Using Semantic Beam Search (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing black-box adversarial attacks on pre-trained models generate adversarials with greedy search. |
| Approach: | They propose an efficient beam search black-box adversarial attack method . they use contextual semantic filtering to effectively reduce the search space . |
| Outcome: | The proposed method shows good performance in terms of attack success rate, number of queries, and semantic similarity for two tasks: NL code search classification and retrieval tasks. |
Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that neural natural language processing models are vulnerable to backdoor attacks. |
| Approach: | They propose to inject neural models with backdoors activated by word substitution . their results raise a serious alarm to the security of NLP models, they argue . |
| Outcome: | The proposed backdoors are activated by a learnable combination of word substitution and exhibit higher invisibility than previous methods. |
MirageBackdoor: A Stealthy Attack that Induces Think-Well-Answer-Wrong Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing CoT backdoor attacks manipulate intermediate reasoning steps to steer the model toward incorrect answers, but these corrupted reasoning traces are readily detected by prevalent process-monitoring defenses. |
| Approach: | They propose a backdoor attack that exploits the model's post-output space to preserve clean CoTs while selectively steering the final answer toward a specific target. |
| Outcome: | Experiments show that MirageBD achieves over 90% success rate across four datasets and five models with a poison ratio of only 5%. |
Contextualized Perturbation for Textual Adversarial Attack (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing techniques for generating adversarial examples are driven by local heuristic rules that are agnostic to the context, resulting in unnatural and ungrammatical outputs. |
| Approach: | They propose a ContextuaLized AdversaRial Example generation model that generates fluent and grammatical outputs through a mask-then-infill procedure. |
| Outcome: | The proposed model outperforms baseline models in terms of attack success rate, textual similarity, fluency and grammaticality. |
PARSE: An Efficient Search Method for Black-box Adversarial Text Attacks (2022.coling-1)
Copied to clipboard
| Challenge: | Neural networks are vulnerable to adversarial examples, i.e., under a black-box scenario. |
| Approach: | They propose a word-level search algorithm that searches for subareas under dynamic search space following the subarea importance. |
| Outcome: | The proposed algorithm can achieve comparable success rates to complex search methods while saving numerous queries and time. |
VIGIL: Defending LLM Agents Against Tool-Stream Injection via Verify-Before-Commit (2026.acl-long)
Copied to clipboard
| Challenge: | Existing defenses for indirect prompt injection are limited by static protection mechanisms . existing models prioritize injected rules due to strict alignment, whereas static protections sever the feedback loop required for adaptive reasoning. |
| Approach: | They propose a framework that shifts the paradigm from restrictive isolation to a verify-before-commit protocol. |
| Outcome: | The proposed framework outperforms state-of-the-art dynamic defenses by reducing the attack success rate by over 22% while more thandoubling utility under attack compared to static baselines. |
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020.emnlp-main)
Copied to clipboard
| Challenge: | Using simulated experiments, we demonstrate that MT systems can be stolen even when imitation models have different input data or architectures than their target models. |
| Approach: | They propose a defense that modifies translation outputs to misdirect optimization of imitation models. |
| Outcome: | The proposed defense degrades the adversary’s BLEU score and attack success rate at some cost in the defender’s performance and inference speed. |
The Mark Fades: Adaptive Evolutionary Paraphrase-based Attack against LLM Watermarks (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing paraphrase-based watermark removal methods struggle to balance efficacy with text quality. |
| Approach: | They propose a training-free evolutionary framework that models watermark removal as a constrained multi-objective optimization problem by using a Pseudo-Log-Likelihood-guided mutation to precisely target and modify watermark-carrying tokens. |
| Outcome: | The proposed method outperforms baseline methods on a Qwen3 series watermark scheme while maintaining high semantic fidelity. |
Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have shown strong potential in various downstream tasks. |
| Approach: | They propose to model adversarial attack task as a sequential decision-making problem where the whole attack process is sequential with two decision- making problems, i.e., word finder and word substitution. |
| Outcome: | The proposed approach achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT. |
SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | a scalable approach to classify text with sensitivity is costly because of exponential time complexity. |
| Approach: | They propose a framework for calculating word-level local and global sensitivities . they use a CHECKLIST-generated sentiment analysis dataset to test their approach . |
| Outcome: | The proposed framework can be used to calculate word-level local and global sensitivities . it improves attacks by 15.58%, while using sensitivity as an additional reward improves . |
BadFair: Backdoored Fairness Attacks with Group-conditioned Triggers (2024.findings-emnlp)
Copied to clipboard
| Challenge: | *BadFair* is a backdoored fairness attack methodology that discriminates and produces incorrect results for specific groups. |
| Approach: | They propose a backdoored fairness attack methodology that discriminates and produces incorrect results for specific groups. |
| Outcome: | The proposed model achieves 85% success rate in attacks aimed at target groups on average while only incurring a minimal accuracy loss. |
Think Twice, Generate Once: Safeguarding by Progressive Self-Reflection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models generate coherent and contextually relevant text, but their deployment raises significant concerns about the potential for harmful or inappropriate content. |
| Approach: | They propose a novel inference-time technique that empowers LLMs to self-monitor and correct their outputs dynamically. |
| Outcome: | The proposed method reduces the attack success rate from 77.47% to 5.86%, to Llama-3.1-8B base from 89.70% to 5.56%, and to Qwen2.5-7B-Instruct from 44.44% to 3.84%, without additional training. |
Analysing the Safety Pitfalls of Steering Vectors (2026.findings-acl)
Copied to clipboard
| Challenge: | Activation steering has emerged as a powerful tool to shape LLM behaviour without the need for weight updates. |
| Approach: | They propose to audit steering vectors obtained with Contrastive Activation Addition (CAA) and propose a mechanistic explanation for this finding. |
| Outcome: | The proposed approach significantly improves the success rate of jailbreak attacks, with stronger amplification under simple template-based attacks. |
MEGen: Generative Backdoor into Large Language Models via Model Editing (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for training large language models are limited to yes-or-no discriminative tasks, leading users to underestimate the potential risks. |
| Approach: | They propose an editing-based generative backdoor that expands the backdoor to generative tasks in a unified format of any text-to-any text. |
| Outcome: | The proposed model achieves high attack success rate by adjusting only a small set of local parameters with few-shot samples. |
P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Defending Large Language Models (LLMs) against backdoors has long been trapped in a "cat-and-mouse" dilemma where defenders passively react to ever-shifting attack strategies. |
| Approach: | They propose a general and effective defense algorithm that implants benign triggers to reshape the model’s decision boundary. |
| Outcome: | The proposed defense algorithm can neutralize malicious backdoors while preserving task performance. |
Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | HKVE selectively accepts gradient optimization results based on the distribution of attention scores across different layers, ensuring that every optimization step positively contributes to the attack. |
| Approach: | They propose a framework that selectively accepts gradient optimization results based on the distribution of attention scores across different layers and selectively takes them into account when calculating the attack success rate. |
| Outcome: | The proposed framework outperforms existing methods by achieving success rates of 75.08% on MiniGPT4, 85.84% on LLaVA and 81.00% on Qwen-VL. |
Locket: Robust Feature-Locking Technique for Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing FLoTEs (e.g., password-locked models) fail to meet these criteria. |
| Approach: | They propose a feature-locking technique that is effective in refusing locked features . they propose scalable FLoTEs that can selectively disable specific features of a model . |
| Outcome: | The proposed solution is effective (100% refusal rate), utility-preserving ( 7% utility degradation), robust (5% attack success rate), and scalable to multiple features and clients. |
Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing red-teaming methods generate adversarial attacks to identify vulnerabilities, but they face slow performance, limited categorical diversity, and high resource demands. |
| Approach: | They propose a method that generates multiple adversarial prompt mutations per iteration and ranks them using scoring functions. |
| Outcome: | The proposed method achieves a 95% attack success rate and reduces time to a 90% ASR by 15.2%. |
PromptAttack: Probing Dialogue State Trackers with Adversarial Prompts (2023.findings-acl)
Copied to clipboard
| Challenge: | Toward building more robust and reliable conversational systems, we introduce a prompt-based learning approach to automatically generate effective adversarial examples to probe DST models. |
| Approach: | They propose a prompt-based learning approach to automatically generate effective adversarial examples to probe DST models. |
| Outcome: | The proposed framework leads to the greatest reduction in accuracy and the best attack success rate while maintaining good fluency and a low perturbation ratio. |
Exploiting Prompt-induced Confidence for Black-Box Attacks on LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are vulnerable to adversarial attacks even in strict black-box settings with only hard-label feedback. |
| Approach: | They propose a black-box framework that leverages prompt-induced confidence as an auxiliary signal to guide attacks. |
| Outcome: | The proposed framework improves the attack success rate and query efficiency while maintaining semantic fidelity. |
BITE: Textual Backdoor Attacks with Iterative Trigger Injection (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to defend against backdoor attacks are based on model stealing, model thieving and training data extraction attacks. |
| Approach: | They propose a backdoor attack that poisons training data to establish strong correlations between the target label and a set of “trigger words” These trigger words are iteratively identified and injected into the target-label instances through natural word-level perturbations. |
| Outcome: | The proposed attack is significantly more effective than baseline methods while maintaining decent stealthiness, raising alarm on the usage of untrusted training data. |
SPIRIT: Patching Speech Language Models against Jailbreak Attacks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Speech language models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. |
| Approach: | They propose post-hoc patching defenses to intervene during inference by modifying the SLM’s activations that improve robustness up to 99% with negligible impact on utility and without any re-training. |
| Outcome: | The proposed defenses improve robustness up to 99% with negligible impact on utility and (ii) without any re-training. |
Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that text-to-image models are vulnerable to adversarial perturbations . |
| Approach: | They investigate the impact of adversarial attacks on different POS tags within text prompts on T2I models. |
| Outcome: | The proposed model is vulnerable to adversarial perturbations with noun perturbations in text prompts. |
Jailbreak LLMs through Internal Stance Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to exploit LLMs' inherent safety mechanism, including GCG and AutoDAN, are ineffective for certain malicious requests. |
| Approach: | They propose a method that generates jailbreak prompts to suppress a refusal stance and induce affirmative responses by modifying adversarial prompts. |
| Outcome: | The proposed method outperforms the best baseline approach in Llama-2-7b-chat and achieves a 92.2% success rate across all models. |
A Curious Case of Searching for the Correlation between Training Data and Adversarial Robustness of Transformer Textual Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that fine-tuned textual transformer models are vulnerable to adversarial text perturbations. |
| Approach: | They extract 13 different features representing a wide range of input fine-tuning corpora properties and use them to predict adversarial robustness of the fine- tuned models. |
| Outcome: | The proposed framework can be used as an additional tool for robustness evaluation since it saves 30x-193x runtime compared to the traditional technique and can be easily used under adversarial training. |
AIP: Subverting Retrieval-Augmented Generation via Adversarial Instructional Prompt (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing RAG attacks rely on manipulating user queries, but exploit instructional prompts to manipulate RAG outputs covertly. |
| Approach: | They propose an attack that exploits adversarial instructional prompts to manipulate RAG outputs . they propose a query generation strategy that simulates realistic linguistic variation in user queries . |
| Outcome: | The proposed attack exploits instructional prompts to manipulate RAG outputs . it achieves up to 95.23% attack success rate while maintaining benign functionality . |
RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate RAG documents require knowledge of the target RAG system’s internal composition and implementation details, whereas black-box methods are unable to utilize interactive information. |
| Approach: | They propose a RIPRAG attack framework that treats the target RAG system as a black box and leverages a Reinforcement Learning from Black-box Feedback (RLBF) method to optimize the generation model for poisoned documents. |
| Outcome: | The proposed method achieves an attack success rate (ASR) improvement of up to 0.72 compared to baseline methods. |
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking (2026.acl-long)
Copied to clipboard
| Challenge: | Existing attacks optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model’s safety-retrieval mechanism. |
| Approach: | They propose a push-pull approach which suppresses attention to system-prompt tokens and anchors generation on adversarial image features to avoid collisions. |
| Outcome: | The proposed approach reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations. |
Defense Against Prompt Injection Attack by Leveraging Attack Techniques (2025.acl-long)
Copied to clipboard
| Challenge: | Recent attacks leverage LLMs’ instruction-following abilities and their inabilities to distinguish instructions injected in the data content. |
| Approach: | They invert the intention of prompt injection methods to develop novel defense methods based on previous training-free attack methods by repeating the attack process with the original input instruction rather than the injected instruction. |
| Outcome: | The proposed methods outperform existing defense approaches, achieving state-of-the-art results. |
CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not noticed the safety risks of large language models . authors evaluated 1,400 questions in multi-turn dialogue coreference . |
| Approach: | They are the first to evaluate LLM safety in multi-turn dialogue coreference . they created a dataset of 1,400 questions and tested five open-source models . |
| Outcome: | The study shows that model safety decreases in multi-turn dialogue coreference scenarios . the highest success rate was with the LLaMA2-Chat-7b model, while the lowest was with mistral-7B-Instruct model . |
Defending against Indirect Prompt Injection by Instruction Detection (2025.findings-emnlp)
Copied to clipboard
Tongyu Wen, Chenglong Wang, Xiyuan Yang, Haoyu Tang, Yueqi Xie, Lingjuan Lyu, Zhicheng Dou, Fangzhao Wu
| Challenge: | Indirect Prompt Injection attacks can be exploited by LLMs that are embedded with external data. |
| Approach: | They propose a detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. |
| Outcome: | The proposed approach reduces the success rate of attacks to 0.03% on the BIPIA benchmark. |
Nullspace Disentanglement for Red Teaming Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work mainly leverages crowd workers to construct test cases. |
| Approach: | They propose a black-box approach that exploits the unique properties of the nullspace to disentangle and regulate the crucial success information within test cases. |
| Outcome: | The proposed approach outperforms baseline methods regarding the attack success rate and excels in aspects of diversity and fluency. |
Adversarial Preference Learning for Robust LLM Alignment (2025.findings-acl)
Copied to clipboard
Yuanfu Wang, Pengyu Wang, Chenyang Xi, Bo Tang, Junyi Zhu, Wenqiang Wei, Chen Chen, Chao Yang, Jingfeng Zhang, Chaochao Lu, Yijun Niu, Keming Mao, Zhiyu Li, Feiyu Xiong, Jie Hu, Mingchuan Yang
| Challenge: | Modern language models rely on Reinforcement Learning from Human Feedback (RLHF) to encourage safe behaviors, but they remain vulnerable to adversarial attacks due to three key limitations: (1) the inefficiency and high cost of human annotation; (2) the vast diversity of potential adversarials; and (3) the risk of feedback bias and reward hacking. |
| Approach: | They propose an iterative adversarial training method that incorporates three key innovations to address these challenges. |
| Outcome: | Experiments on Mistral-7B-Instruct-v0.3 show that the proposed method significantly enhances robustness and reduces harmful outputs from 5.88% to 0.43%. |
DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. |
| Approach: | They propose an end-to-end generative approach for jailbreak rewriting inspired by diffusion models that uses a sequence-tosequence (seq2sequ) diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. |
| Outcome: | Experiments on Advbench and Harmbench show that the proposed method outperforms autoregressive jailbreak models across evaluation metrics including ASR, fluency, diversity and diversity. |
Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can lead to undesired consequences when misaligned with human values . previous studies have shown misalignment of LLMs with human value using expert-designed or agent-based emulated bias scenarios . |
| Approach: | They investigate whether large language models (LLMs) are misaligned with human values . they find no significant differences in understanding of HVSB between LLMs . |
| Outcome: | The results show that large language models do not have lower misalignment rates and attack success rates . the study also shows that smaller language models have the ability to explain HVSB . |
Experience-Driven Multi-Agent Optimization for Black-Box Jailbreak Attacks on Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for jailbreak have poor transferability and high sensitivity to preprocessing . EMJO provides an effective and scalable paradigm for systematic jailbreak optimization . |
| Approach: | They propose a model that couples agents into a closed-loop "probe–evaluate–revise” process . they propose EMJO, which can be query-efficient and transferable, under black-box access. |
| Outcome: | a new approach outperforms existing jailbreak baselines on diverse LLMs . it achieves up to 11% improvement in attack success rate while reducing query cost . |
NEXUS: Network Exploration for eXploiting Unsafe Sequences in Multi-Turn LLM Jailbreaks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to large language models suffer from limited exploration of adversarial space . multi-turn jailbreaks that distribute malicious intent across benign exchanges are vulnerable . NEXUS aims to exploit the adversarials of LLMs for maximum effectiveness in jailbreak scenarios . |
| Approach: | They propose a framework for constructing, refining, and executing optimized multi-turn attacks . NEXUS builds a semantic network of thought that captures a comprehensive representation of the adversarial space . |
| Outcome: | NEXUS can achieve higher attack success rate than state-of-the-art approaches . it builds a semantic network of thought that captures a comprehensive representation of the adversarial space . |
SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have increased their vulnerability to adversarial manipulation. |
| Approach: | They propose to target audio-only adversarial attacks on multimodal audio–video–language models . they show that attacks can be successful at low perceptual distortions . |
| Outcome: | The proposed models achieve up to 96% success rate under realistic conditions . the proposed models are more robust to noise than to noise and distortion than to speech recognition systems . |
Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in multimodal systems have demonstrated remarkable capabilities in generating multimodal content from multimodal inputs. |
| Approach: | They propose a benchmark that leverages large language models to generate deceptive text samples to exploit compositional vulnerabilities across different modalities. |
| Outcome: | The proposed approach exploits compositional vulnerabilities across images, videos, and audios. |
Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities but their misuse for harmful purposes remains a concern. |
| Approach: | They propose a jailbreaking technique that exploits weaknesses in LLMs' architecture . they propose abductive framing and symbolic encoding to bypass safeguards . |
| Outcome: | The proposed technique achieves over 95% attack success rate on GPT-series models and 70% across all targets. |
Red Queen: Exposing Latent Multi-Turn Risks in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing jailbreak strategies are single-turn with explicit malicious intent, failing to reflect real-world scenarios where interactions can be multi-turn and users can conceal intents. |
| Approach: | They propose a jailbreak attack that constructs a multi-turn scenario and conceals malicious intent under the guise of preventing harm. |
| Outcome: | The proposed attack achieves 87.6% attack success rate (ASR) on GPT-4o and 77.1% on Llama3-70B compared to previous jailbreak attacks . |
STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red Teaming (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to jailbreak prompts that can elicit harmful or inappropriate responses. |
| Approach: | They propose a black-box framework for automated red teaming that integrates a Multi-Agent System with a Strategy-Response Multiplex Network and employs network-driven optimization to sample effective attack strategies. |
| Outcome: | The proposed framework surpasses existing methods and achieves higher attack success rate (ASR) at lower computational cost. |
Unveiling Vulnerability of Self-Attention (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies focus on manipulating word inputs, but they lack generalization to versatile real-world attacks. |
| Approach: | They propose a powerful perturbation technique which perturbs the attention scores within the SA matrices via meticulously crafted attention masks. |
| Outcome: | The proposed perturbation technique achieves high attack success rate (98%) and low cost. |
MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Knowledge Poisoning Attacks (2026.acl-long)
Copied to clipboard
Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dimitrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-Wei Chang, Daniel Kang, Heng Ji
| Challenge: | Existing research exposes multimodal large language models to knowledge poisoning attacks . localized poisoning attack achieves up to 56% success rate even under restricted access . globalized poison attack completely disrupts model generation to 0% accuracy with just one poisoned content. |
| Approach: | They propose a framework to study the vulnerability of multimodal RAG under knowledge poisoning attacks. |
| Outcome: | The proposed framework exploits two new attack strategies on multimodal RAGs under knowledge poisoning. |
PRA-RAG: Provably Robust Aggregation in Retrieval-Augmented Generation against Retrieval Corruption (2026.findings-acl)
Copied to clipboard
Xue Tan, Yi Zheng, Chang Huo, Yunruo Zhang, Yu Liu, Hao Luan, Zhuyang Yu, Jun Dai, Xiaoyan Sun, Ping Chen
| Challenge: | Existing defense mechanisms lack theoretical robustness guarantees and perform unreliably when the LLM has limited knowledge of the retrieved content. |
| Approach: | They propose a provably robust retrieval aggregation algorithm designed to defend against poisoning attacks on retrieved texts. |
| Outcome: | Experiments show that PRA-RAG reduces the attack success rate to as low as 1% while maintaining an accuracy of 71%, significantly outperforming representative state-of-the-art (SOTA) methods. |
LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety (2026.acl-long)
Copied to clipboard
Junxiao Yang, Haoran Liu, Jinzhe Tu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Jiaqi Weng, Jialing Tao, Hui Xue, Hongning Wang, Han Qiu, Minlie Huang
| Challenge: | Large language models (LLMs) have demonstrated better safety performance in high-resource languages than in low-resourced languages. |
| Approach: | They propose language-agnostic semantic alignment (LASA) which anchors safety alignment directly in semantic bottlenecks. |
| Outcome: | The proposed approach significantly improves safety across all languages: average attack success rate drops from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and remains within 3–4% across Qwen2.5 and Qwend3 Instruct models (7B–32B). |
Defending LLMs against Jailbreak Attacks via Template-Based ICL with a Defensive Suffix (2026.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art large language models (LLMs) are vulnerable to jailbreak attacks, such as GCG and AutoDAN. |
| Approach: | They propose to take the advances of online In-Context Learning and an offline defensive suffix and optimize them using an iterative algorithm and an online stochastic random search to identify the most effective ICL demonstrations. |
| Outcome: | The proposed method reduces attack success rate to nearly *0% while maintaining the model’s utility on benign tasks and incurring only *negligible* computational overhead. |
More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful but weak when inputs are perturbed. |
| Approach: | They evaluate LLMs that are more powerful than single LLM in math question answering . they use a unified sampling-and-voting framework to evaluate their models . |
| Outcome: | The proposed models show that collaboration between agents improves accuracy and clean accuracy even with a large number of agents. |